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SECTION  A.  Background 

1.  PURPOSE  This  instruction  identifies  a  best- 

practice  process  for  AFFTC  organizations  for 
evaluating  software  Operational  Flight  Program  (OFP) 
and/or  associated  hardware  changes.  Associated 
hardware  covers  line  replaceable  units  (such  as 
computers/processors/sensors/displays)  that 

interface  with  the  software.  It  is  applicable  to  all 
organizations  involved  in  test  programs,  including 
tenants  and  contractors. 

2.  APPLICABILITY.  New  test  programs  should 
incorporate  as  much  of  this  process  into  their 
structure  as  possible,  and  existing  test  forces  should 
adapt  as  much  of  it  as  possible  into  their  existing 
organization.  The  AFFTC  representative  for  new  test 
projects  should  have  the  System  Program  Office 
(SPO)  contract  for  as  much  of  this  process  as 
possible.  In  addition,  this  process  may  be  used  to 
evaluate  an  initial  release  of  an  OFP. 

3.  FACTORS  AFFECTING  TEST  FORCE 
APPROACH  TO  FLIGHT  TEST  OF  SOFTWARE 
CHANGES.  Although  a  test  force  should  endeavor 
to  implement  as  much  of  this  process  as  possible, 
funds  and  schedule  are  not  unlimited  in  any  test 
program.  Hence  there  will  be  tradeoffs  in  the 
implementation  of  this  process.  Some  things  to 
consider  when  making  these  hard  decisions  are: 

3.1.  Flight  Critical  Vs.  NonFlight  Critical  Software. 

Flight  critical  software  affects  the  guidance  and/or 
control  of  the  vehicle,  whereas  nonflight  critical  does 
not.  Due  to  the  immense  integration  of  processors 
and  software  on  modern  aircraft,  the  lines  are  blurred 
between  flight  critical  and  nonflight  critical  software. 
Flight  critical  encompasses  systems  that  if  not 


operating  correctly,  or  if  shut  down,  would  jeopardize 
safety  of  flight.  Some  examples  of  flight  critical 
software  are  flight  control  systems,  terrain  following 
systems  and  terrain  avoidance  systems.  One  might 
think  that  an  air-to-ground  radar  is  a  nonflight  critical 
system.  However,  if  it  were  to  provide  ranging  data  to 
a  terrain  avoidance  system,  it  must  be  considered 
flight  critical  at  least  for  that  testing.  Levels  of 
redundancy  and  built  in  tests  need  to  also  be  factored 
in.  Each  program  needs  to  assess  and  determine 
which  systems  fit  into  which  category. 

3.2.  Type  of  Test  Program.  Implementation  of  the 
OFP/hardware  evaluation  process  in  a  small  proto¬ 
type/research  program  is  relatively  easier  than  in  a 
large  production  program  because  of  the  closeness  of 
the  test  team  members.  Implementation  in  a  large 
production  program  is  difficult  because  of  the  diverse 
population  and  large  inertia  that  makes  flow  of 
information  more  difficult.  Large  programs  need  to  be 
aware  of  this  and  fight  to  overcome  it. 

3.3.  Maturity  or  Phase  of  Test  Program.  As  a 

program  progresses  and  becomes  more  mature,  a 
conscious  decision  to  lessen  some  of  the 
requirements  and  formality  of  this  process  may  be 
made.  However,  the  test  team  must  continually 
evaluate  this  and  be  alert  for  traps  waiting  to  catch 
them. 

4.  TERMINOLOGY.  Each  "Best  Practice"  will  be 
identified  by  criticality  using  the  following  modifiers: 

4.1.  Essential:  Practices  identified  as  essential  are 
either  required  by  regulations  or  are  considered 
critical  to  the  AFFTC  role  in  the  evaluation  of 
software  OFP/hardware  changes. 
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4.2.  Highly  Recommended:  Practices  that  have  been 
shown  to  have  the  highest  payoff  potential  for 
contributing  to  the  AFFTC  role  in  the  evaluation  of 
software/OFP  hardware  changes. 

4.3.  Good  Ideas:  Practices  that  have  been  shown  to 
have  some  payoff  potential  but  that  may  have  a 
relatively  high  cost-benefit  ratio  in  terms  of  utilization 
of  resources  such  as  manpower  and  laboratories. 


SECTION  C.  AFFTC  Participation  in  Primarily 
Contractor  Processes.  This  section  describes  the 
processes  that  are  normally  conducted  by  the 
contractor,  but  it  is  essential  that  the  AFFTC  test 
force  members  participate.  The  more  participation  up 
front  in  the  process,  the  better  the  evaluation  will  be. 
By  so  doing,  when  it  comes  time  to  test,  the  test  force 
will  be  better  able  to  ensure  the  OFP  change 
addresses/corrects  the  root  problem  identified  in  the 
discrepancy  or  change  requirement.  The  following 
paragraphs  provide  participation  opportunities  for  the 
AFFTC. 


SECTION  B.  Software  OFP  and/or  Associated 
Hardware  Update  Evaluation  Process.  Figure  1  is  a 
general  flow  diagram  of  the  process.  Each  block  will 
be  addressed  below.  Note  that  the  process  can  be 
very  iterative,  and  the  numerous  "inner  feedback 
loops"  are  not  shown.  That  is,  anywhere  in  the 
process,  if  problems/anomalies  are  encountered,  you 
may  have  to  go  back  or  digress  to  another  point  and 
continue  the  process  from  that  point. 


5.  DEFICIENCY  REPORT  (DR).  It  is  essential  that 
the  USAF  Deficiency  Report  (DR)  system  be  used  as 
the  formal  input  through  the  SPO  to  the  contractor 
software  update  process.  This  process  was  most 
recently  known  as  the  Product  Quality  Deficiency 
Report  (PQDR)  system  and  prior  to  that  it  was  known 
as  the  Service  Report  (SR)  system.  A  DR  can  be 
written  at  anytime,  but  typically  is  written  when  it  is 
apparent  the  contractor  does  not  intend  to  fix  the 
discrepancy.  At  the  completion  of  testing, 
outstanding  issues  are  reviewed  and  DRs  are  written 
as  necessary. 
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7.  CONTRACTOR  DISCREPANCY  PROCESS.  It 
is  a  good  idea  to  have  some  informal  method  for 
correlating  the  USAF  DR  inputs  with  the  contractor's 
internal  discrepancy  system.  This  will  ensure  that 
testers  can  trace  the  status  and  resolution  of  their 
DRs.  It  is  highly  recommended  that  AFFTC 
engineers  be  aware  of  the  discrepancies  being 
identified  and  documented  by  the  contractor  in  their 
internal  discrepancy  system.  This  can  be 
accomplished  at  a  variety  of  levels  depending  on  how 
closely  the  AFFTC/contractor  team  normally 
coordinate  on  a  given  project.  Some  contractors  are 
very  reluctant  to  divulge  these  internal  discrepancies, 
and  SPO  involvement  may  be  required  to  obtain  it. 
Knowledge  of  these  discrepancies  can  prevent 
wasted  testing  and  increase  flight  test  safety. 

8.  NEW  USER  REQUIREMENT/SPECIFIC  CHANGE 
REQUESTS.  It  is  highly  recommended  that  AFFTC 
engineers  be  aware  of  the  content  and  status  of 
Engineering  Change  Proposals  (ECPs)  and/or 
Software  Change  Requests  (SCRs)  that  are  being 
developed  between  the  SPO  and  the  contractor.  This 
may  be  the  first  time  the  AFFTC  will  be  aware  of  the 
impact  of  user  change  requirements.  It  is  a  good  idea 
for  AFFTC  engineers  to  review  and  participate  in  the 
ECP/  SCR  process. 

9.  PDR/CDR  "AT  BUILD"  LEVEL.  It  is  highly 
recommended  that  AFFTC  engineers  take  advantage 
of  any  opportunities  to  participate  in  the  contractor's 
development  process  leading  up  to  a  design  including 
analysis  and  testing.  Participation  in  these  design 
stages  allows  the  AFFTC  to  influence  the  design  so 
that  we  have  a  higher  quality  product  to  test  later.  It 
is  highly  recommended  that  AFFTC  engineers  be 
aware  of  the  results  of  the  Preliminary  Design 
Reviews  (PDRs)  and  Critical  Design  Reviews  (CDRs) 
for  each  software  release  or  "build."  This  is  the  first 
time  that  a  variety  of  individual  changes  are  brought 
together  in  a  package.  It  is  a  good  idea  for  AFFTC 
engineers  to  participate  in  the  PDR/CDR  process. 

10.  SOFTWARE  CODING.  Little  or  no  AFFTC 
involvement  is  required/desired  during  this  stage 
where  the  contractor  is  translating  PDR/CDR 
requirements  into  lines  of  software  code. 

11.  CONTRACTOR  QUALIFICATION  TESTING.  It 
is  highly  recommended  for  AFFTC  engineers  to  be 


aware  of  what  the  contractor's  software  Verification 
and  Validation  (V&V)/Systenr  Integration  Lab 
(SIL)/Lab  Testing  process  and  qualification  test 
process  is.  It  is  highly  recommended  that  AFFTC 
engineers  participate  (at  an  appropriate  level)  in  the 
contractor's  qualification  testing.  In  order  to 
accomplish  this,  the  test  force  needs  to  know  what 
testing  is  being  done  and  when.  Minimal  (or  no) 
AFFTC  participation  is  recommended  for  contractor 
testing  at  the  module  level.  As  the  testing  proceeds 
to  more  integrated  system  level  testing,  AFFTC 
participation  is  more  beneficial.  In  particular,  any 
testing  identified  by  the  contractor  as  flight 
qualification  testing  has  the  highest  payoff  potential 
for  AFFTC  participation.  Flight  qualification  testing 
usually  includes  man  and  hardware  in  the  loop  testing 
near  the  end  of  the  test  cycle  just  prior  to  release  for 
flight.  It  is  highly  recommended  that  AFFTC  test 
force  personnel  be  aware  of  what  software/hardware 
configuration  differences  exist  between  the  lab  and 
the  test  article.  Failure  Modes  and  Effects  Testing 
(FMET)  also  has  a  high  payoff  potential  for  AFFTC 
participation.  If  participation  is  not  possible,  a 
synopsis  briefing/report  from  the  contractor  is  highly 
recommended  in  place  of  version  description 
documents/detailed  report. 

SECTION  D.  Combined  AFFTC  and  Contractor 
Process 

12.  COMBINED  QUALIFICATION  TESTING. 

Facilities  that  exist  at  AFFTC  such  as  the  Integration 
Facility  for  Avionics  System  Testing  (IFAST), 
Benefield  Anachoic  Facility  (BAF),  Flight  Test 
Avionics  Lab  (FTAL),  and  Test  and  Evaluation 
Mission  Simulation  (TEMS)  provide  the  opportunity 
for  combined  testing  and  evaluation  of  software  OFP 
and/or  associated  hardware  changes  prior  to  and 
during  flight  test.  In  some  cases  these  facilities  are 
"run"  by  the  contractor  with  AFFTC  participation.  In 
other  cases  these  facilities  are  "run"  by  the  AFFTC 
with  little  or  no  contractor  participation.  It  is  highly 
recommended  that  these  facilities  be  used  by  both  the 
AFFTC  and  contractor  to  maximize  the  effectiveness 
of  the  upcoming  flight  testing  of  a  particular  software 
change.  There  are  a  wide  range  of  options  for 
accomplishing  this  that  are  detailed  in  other 
documents,  such  as  the  Avionics  Test  and 
Integration  (ATIC)  Information  Sheet  published  by 
ATIC. 
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12.1  Another  type  of  combined  AFFTC  and 
contractor  testing  that  is  highly  recommended  and 
has  been  used  effectively  on  several  projects  is 
"Confidence  Testing.”  The  purpose  of  these  tests  is 
to  provide  CTF  management  with  additional 
confidence  that  a  given  software  release  is  indeed 
ready  for  flight  test.  These  tests  can  be  conducted 
either  at  the  EAFB  facilities  previously  described  or  at 
the  contractor's  main  facilities.  These  tests  are 
usually  conducted  after  all  of  the  "normal"  testing  has 
been  completed  but  before  the  software  is  approved 
for  flight  test.  The  primary  difference  between 
confidence  testing  and  all  of  the  previous  testing  is 
that  the  confidence  test  plan  is  produced  by  flight 
testers  as  opposed  to  designers.  This  tends  to  give 
the  testing  a  different  "slant"  that  can  often  uncover 
discrepancies  that  were  not  observed  by  the  normal 
test  process.  The  flight  testers  contributing  to  this 
test  plan  can  be  both  AFFTC  and  contractor 
engineers  and  pilots. 

SECTION  E.  AFFTC  Process.  This  section  describes 
the  process  that  is  normally  conducted  by  the  AFFTC 
members  of  the  test  force  but  usually  also  includes 
participation  by  the  contractor  members  of  the  test 
force. 

13.  TECHNICAL  INFORMATION  ACQUISITION. 
It  is  highly  recommended  that  the  worker  level  test 
force  personnel  gather  technical  information  on  what 
the  changes  are  at  the  build  level  in  order  to  present 
the  information  to  test  force  management  personnel. 
A  good  idea  to  accomplish  this  is  for  the  designers 
and  testers  to  sit  together  and  do  an  informal  "walk 
through”  design  level  description  of  the  change,  the 
reason  for  the  change,  what  is  expected  from  the 
change,  and  its  compatibility  with  other 
software/hardware.  This  is  the  first  opportunity  for 
the  tester  to  start  defining  the  actual  testing  that  will 
be  necessary.  A  normal  product  of  this  phase  is  a 
version  description  document  from  the  contractor, 
which  may  or  may  not  have  value.  It  is  highly 
recommended  that  the  test  force  find  out  what  the 
contractor's  general  V+V  process  is,  and  it  is  a  good 
idea  to  know  exactly  what  V+V  the  specific  OFP  will 
go  through. 

14.  PRELIMINARY  TEST  FORCE  MANAGEMENT 
REVIEW.  Around  the  time  of  the  software/hardware 
CDR,  it  is  highly  recommended  that  the  test  force 
conduct  a  management  review  to  ensure  awareness  of 
upcoming  software/hardware  changes  and  schedules, 
so  that  flight  test  implications  can  be  assessed. 

15.  FLIGHT  TEST  PLANNING. 


15.1.  It  is  highly  recommended  that  flight  test 
planning  begin  soon  after  a  software/hardware 
change  is  being  solidified  (PDR/CDR).  It  is  essential 
that  the  impacts  of  the  software  change  on  flight 
testing  be  identified  and  planned.  These  impacts 
include  aspects  such  as  test  conditions  and 
procedures,  operating  limitations,  and  emergency 
procedures,  instrumentation,  range  requirements,  and 
safety. 

15.2.  Regression  flight  testing  means  different  things 
to  different  people.  Usually  regression  flight  testing 
means  repeating  a  test  that  has  already  been 
conducted  with  a  previous  version  of  the  software. 
Usually  the  majority  of  those  repeat  tests  are  aimed 
specifically  at  the  areas  where  changes  are  expected 
(hopefully  improvements).  Sometimes  regression 
flight  testing  includes  a  "core”  set  of  tests  that  are 
repeated  every  time  a  software  change  is  made  even 
when  that  change  is  not  expected  to  affect  the  areas 
in  the  core  test.  Core  tests  may  be  appropriate  when 
there  is  uncertainty  in  the  software  mechanization  and 
testing  process  (or  a  history  of  unexpected  results). 
Core  tests  may  not  be  needed  in  flight  if  analysis 
shows  that  software  change  impacts  can  be  limited  to 
certain  areas  or  functions,  such  as  redundancy 
management  and  modifications  to  executive  code.  If  a 
software  change  dictates  an  entirely  new  test 
condition  or  procedure  that  had  not  been  previously 
flight  tested,  then  that  new  test  cannot  be  considered 
regression  testing,  and  associated  test  planning  and 
review  will  be  required. 

15.3.  It  is  highly  recommended  that  the  core 
regression  test  be  defined  in  the  original  OFP  test 
planning.  By  so  doing,  the  test  force  can  alleviate  its 
requirement  to  re-test  plan  these  core  tests  when  a 
new  OFP  or  associated  hardware  is  released. 

15.4.  Both  a  technical  review  and  safety  review  are 
necessary  as  required  IAW  current  AFFTC 
regulations.  Each  change  will  be  reviewed 
individually  to  determine  if  it  is  within  scope  of  the 
original  planning.  If  regression  test 

ing  has  been  identified  and  planned  adequately  in  the 
original  test  planning/technical  review/safety  review, 
no  further  action  may  be  required.  If  OFP  change 
specific  testing  beyond  this  originally  defined 
regression  testing  is  required,  the  test  team  must 
identify  the  change  -  specific  regression  matrix. 
Technical  Review  Board  (TRB)  and  Safety  Review 
Board  (SRB)  reconvenes  are  more  likely  to  be  needed 
for  flight  critical  changes  than  for  non-flight  critical 
changes.  It  may  not  be  required  to  reconvene  these 
boards  but  it  is  likely  that  a  paper  trail  (AFFTC 
Forms5232b  and  5028  Amendment)  will  be  needed 
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to  document  the  approach  to  flight  testing  the 
software  change.  If  a  new  test  plan  is  generated,  full 
technical  and  safety  reviews  will  be  required.  It  is 
highly  recommended  that  the  Test  Force  produce  a 
consistent  framework  for  the  software  change  process 
so  that  judgments  on  the  proposed  test  approach  will 
be  easier  to  make.  This  consistent  framework  should 
include  a  clear  philosophy  for  "regression”  flight 
testing. 

15.5.  There  has  been  confusion  in  the  past  about 
what  safety  risk  level  is  appropriate  for  regression 
flight  testing.  One  philosophy  is  to  use  the  same  risk 
level  as  the  original  test.  This  philosophy  has  been 
applied  when  the  original  test  was  considered  a 
hazardous  "envelope  expansion"  test.  Another 
philosophy  that  may  be  appropriate  is  that  most  of 
the  "risk”  was  assumed  when  the  original  test  was 
conducted  and  that  the  repeat  test  is  therefore  at  a 
lower  risk  level.  Either  philosophy  may  be  used  as 
long  as  it  is  applied  consistently  and  clearly  identified 
in  the  safety  paperwork. 

16.  IDENTIFY  IMPACTS  OF  QUAFIFICATION 
TEST  RESUETS.  It  is  considered  essential  that  the 
flight  test  impacts  of  all  qualification  test  results  (such 
as  lab,  SIL,  V+V,  IF  AST,  TEMS,  etc.)  be  identified  and 
that  those  impacts  be  considered  in  flight  test 
planning  and  briefed  to  the  key  flight  test 
participants.  A  given  software  change  may  not 
function  as  originally  intended  and  that  unplanned 
effect  may  impact  flight  test  safety  and  efficiency. 
These  unplanned  effects  may  not  be  considered 
significant  (from  a  specification  viewpoint)  by  the 
qualification  testers,  but  they  may  not  be  fully 
cognizant  of  flight  test  impacts.  These  impacts  could 
be  changes  in  test  procedures,  operating  limits  (Flight 
Operating  Limits  Document  (FOLD),  Airframe  and 
Engine  Operating  Limit  (AEOL),  Aircraft  Operating 
Limit  (AOL),  etc.),  or  emergency  procedures. 
Simulation/lab  fidelity  should  be  considered  when 
analyzing  test  results.  Dedicated  reviews  of  the 
qualification  test  results  of  the  new  software  by  flight 
testers  are  highly  recommended  in  order  to  increase 
the  probability  of  identifying  flight  test  impacts.  This 
review  could  range  from  a  formal  meeting  to  a 
sequential  desk  top  review. 

17.  FINAL  TEST  FORCE  MANAGEMENT  REVIEW. 

17.1.  Prior  to  installation  of  new  software/hardware 
on  the  test  vehicle  it  is  essential  that  the  test  force 
complete  a  formal  management  review  process  that 
has  cross  discipline  coordination  to  assess  the  overall 
readiness  to  begin  flight  testing  of  that 
software/hardware.  At  test  forces  with  less  mature 
test  vehicles  this  review  is  normally  part  of  the  test 


force  Configuration  Control  Board  (CCB).  At  test 
forces  with  more  mature  test  vehicles  this  review  may 
be  conducted  by  the  members  of  the  CCB  without  a 
formal  convene  of  the  board.  Additionally,  this  CCB 
authority  may  have  been  delegated  to  the  test  force 
by  the  AFFTC  CCB.  This  review  process  should 
consider  topics  such  as: 

—  Overall  software/hardware  change  summary 
—  Qualification  test  results  and  impacts 
—  Open  discrepancies 
—  Flight  test  planning 

—  Test  procedure  impacts 
—  Technical  and  Safety  Review  results 
—  Ground  test  prerequisites 
—  Specific  impacts  on  operating  limitations  and 
emergency  procedures 
—  Paperwork  status 

—  Other  software/hardware  that  is  compatible/ 
incompatible  with  the  test  OFP/hardware 

17.2.  If  a  formal  CCB  is  convened,  it  is  a  good  idea  to 
have  the  designers  and/or  qualification  testers 
present  the  software  change  summary,  qual  test 
results,  and  open  discrepancies.  It  is  highly 
recommended  that  either  the  contractor  or  AFFTC 
flight  testers  present  the  topics  on  flight  test  planning 
and  on  operating  limitations  and  emergency 
procedures  impacts. 

17.3.  Much  of  the  information  needed  to  support  this 
review  is  not  usually  available  until  very  near 
installation  of  the  new  software  on  the  aircraft  and 
right  before  flight  test.  Therefore  much  of  the 
"homework"  and  preparation  for  flight  test  should 
already  be  completed.  Any  open  items  should  be 
identified  at  the  review  and  a  closure  plan  established. 
One  of  the  main  goals  of  the  review  should  be  to 
determine  if  anything  has  "slipped  through  the 
cracks." 

18.  AIRCREW  TECHNICAL  ORIENTATION.  It  is 
highly  recommended  that  a  dedicated  briefing  to  the 
aircrew  be  conducted  just  prior  to  flight  testing  with 
the  new  software.  This  briefing  should  cover  many  of 
the  same  topics  as  the  management  review  but  with 
special  emphasis  on  the  aircrew  perspective  versus 
the  programmatic  perspective.  It  is  highly 
recommended  that  this  briefing  include  as  many  of 
the  potential  flight  test  crewmembers  as  is  feasible.  A 
highly  recommended  alternative  to  a  briefing  is  a 
written  summary  that  is  made  a  Unit  Flight  Crew 
Information  File  (FCIF)  item.  It  is  essential  that  the 
first  aircrew  to  test  new  software  be  aware  of  all 
changes  with  the  potential  for  impacts  perceivable  by 
that  aircrew. 
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19.  FLIGHT  PREPARATION. 

19.1.  Installation  checks  and  on-aircraft  ground  tests 
are  usually  conducted  as  a  normal  part  of  flight 
preparation  for  new  software/hardware,  such  as 
checksums/configuration  verification.  It  is  highly 
recommended  that  the  management  review  determine 
if  these  checks  and  tests  are  both  necessary  and 
sufficient. 

19.2.  Simulator  dry  runs  are  highly  recommended 
prior  to  flight  test  of  new  software.  These  dry  runs 
are  different  than  confidence  testing  in  that  the  actual 
aircrew  planned  for  the  test  should  accomplish  the 
dry  run  whereas  any  qualified  aircrew  may  conduct 
the  confidence  testing.  It  may  be  possible  to  combine 
the  confidence  test  and  the  dry  run  if  the  aircrew  is 
the  same.  The  dry  runs  may  be  conducted  either  at 
the  contractor's  simulation  facility  or  at  an  AFFTC 
facility. 

19.3.  It  is  essential  that  the  flight  briefing  emphasize 
the  primary  impacts  of  new  software  and  the  expected 
results.  This  briefing  will  be  much  more  effective  if  a 
more  detailed  pilot  briefing  had  been  conducted 
previously.  It  is  essential  that  the  OFP  ID/checksum 
be  included  on  the  test  cards  and  that  the  aircrew 
check  that  the  correct  OFPs  are  loaded  prior  to  flight. 
It  is  highly  recommended  that  the  test  force  build  a 


matrix  to  show  what  OFPs/hardware  are 
compatible/incompatible  with  other  hardware/OFPs  to 
aid  in  the  preflight  preparation.  It  is  highly 
recommended  that  a  one  or  two  page  summary  of  the 
new  software  be  included  with  the  test  cards  and  that 
regression  testing  be  explicitly  identified.  If 
regression  testing  must  be  completed  before 
continuing  on  with  other  test  cards  that  should  also 
be  clearly  identified. 

20.  FLIGHT  TESTING.  Actual  flight  testing  of  new 
software  should  be  reasonably  safe  and  effective  if  all 
of  the  essential  practices,  most  of  the  highly 
recommended  practices,  and  some  of  the  good  ideas 
are  followed.  Unexpected  results  may  still  occur  but 
at  least  a  reasonable  effort  will  have  been  made  to 
minimize  the  likelihood  or  impact  of  those  unexpected 
results.  These  unexpected  results  may  drive  the  test 
team  back  into  this  OFP/Hardware  Evaluation  Process 
at  any  point  to  "reaccomplish"  the  process. 

21.  OT&E  CERTIFICATION/FLIGHT  MANUAL 
INPUTS.  It  is  highly  recommended  that  any  special 
constraints  imposed  by  new  software  on  operational 
testing  be  clearly  identified  in  the  OT&E  Certification 
documentation.  It  is  essential  that  any  unique 
impacts  of  new  software  be  documented  in  the  flight 
manual  for  use  by  operational  crews. 


RICHARD  L.  ENGEL,  Brigadier  General,  USAF 
Commander 


